{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c4c3dffb-98fa-4d14-98ef-80c38d290e1b",
   "metadata": {},
   "source": [
    "# Codex performance on GSM8K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2f7c15ca-974a-49aa-8212-d9b6730df39a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "import re\n",
    "import time\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "from tqdm import tqdm\n",
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b63ddea4-8a1a-4629-9aa6-efdd47bf5e4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "openai.api_key = \"sk-\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6032026c-70fc-4ba8-b353-fd3ecf78eb00",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gsm8k = load_dataset('gsm8k', 'main')\n",
    "validation_index = np.load('../gsm8k/lib_prompt/validation_index.npy')\n",
    "validation_data = gsm8k['train'].select(validation_index)\n",
    "gsm8k_test = gsm8k['test']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "99da2222-207e-4d78-91a6-527af94c9afd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsm8k['train'][0]['question']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0aebf3f1-ff3a-458d-bdd4-42357aafaf20",
   "metadata": {},
   "outputs": [],
   "source": [
    "gsm8k_test = gsm8k['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de555fd-ba2c-4cfa-993f-8b35c01010c6",
   "metadata": {},
   "source": [
    "# Complex Prompt, Acc 66.6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ca22a6a6-73b4-49dd-bf78-1e7b8d93c58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_complex = open('../gsm8k/lib_prompt/prompt_hardest.txt').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "326b98c1-5b9b-41c2-a15b-f676a6814a7a",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?\n",
      "Let's think step by step\n",
      "Angelo and Melanie think they should dedicate 3 hours to each of the 2 chapters, 3 hours x 2 chapters = 6 hours total.\n",
      "For the worksheets they plan to dedicate 1.5 hours for each worksheet, 1.5 hours x 4 worksheets = 6 hours total.\n",
      "Angelo and Melanie need to start with planning 12 hours to study, at 4 hours a day, 12 / 4 = 3 days.\n",
      "However, they need to include time for breaks and lunch. Every hour they want to include a 10-minute break, so 12 total hours x 10 minutes = 120 extra minutes for breaks.\n",
      "They also want to include 3 10-minute snack breaks, 3 x 10 minutes = 30 minutes.\n",
      "And they want to include 30 minutes for lunch each day, so 120 minutes for breaks + 30 minutes for snack breaks + 30 minutes for lunch = 180 minutes, or 180 / 60 minutes per hour = 3 extra hours.\n",
      "So Angelo and Melanie want to plan 12 hours to study + 3 hours of breaks = 15 hours total.\n",
      "They want to study no more than 4 hours each day, 15 hours / 4 hours each day = 3.75\n",
      "They will need to plan to study 4 days to allow for all the time they need.\n",
      "The answer is 4\n",
      "\n",
      "Question: Mark's basketball team scores 25 2 pointers, 8 3 pointers and 10 free throws.  Their opponents score double the 2 pointers but half the 3 pointers and free throws.  What's the total number of points scored by both teams added together?\n",
      "Let's think step by step\n",
      "Mark's team scores 25 2 pointers, meaning they scored 25*2= 50 points in 2 pointers.\n",
      "His team also scores 6 3 pointers, meaning they scored 8*3= 24 points in 3 pointers\n",
      "They scored 10 free throws, and free throws count as one point so they scored 10*1=10 points in free throws.\n",
      "All together his team scored 50+24+10= 84 points\n",
      "Mark's opponents scored double his team's number of 2 pointers, meaning they scored 50*2=100 points in 2 pointers.\n",
      "His opponents scored half his team's number of 3 pointers, meaning they scored 24/2= 12 points in 3 pointers.\n",
      "They also scored half Mark's team's points in free throws, meaning they scored 10/2=5 points in free throws.\n",
      "All together Mark's opponents scored 100+12+5=117 points\n",
      "The total score for the game is both team's scores added together, so it is 84+117=201 points\n",
      "The answer is 201\n",
      "\n",
      "Question: Bella has two times as many marbles as frisbees. She also has 20 more frisbees than deck cards. If she buys 2/5 times more of each item, what would be the total number of the items she will have if she currently has 60 marbles?\n",
      "Let's think step by step\n",
      "When Bella buys 2/5 times more marbles, she'll have increased the number of marbles by 2/5*60 = 24\n",
      "The total number of marbles she'll have is 60+24 = 84\n",
      "If Bella currently has 60 marbles, and she has two times as many marbles as frisbees, she has 60/2 = 30 frisbees.\n",
      "If Bella buys 2/5 times more frisbees, she'll have 2/5*30 = 12 more frisbees.\n",
      "The total number of frisbees she'll have will increase to 30+12 = 42\n",
      "Bella also has 20 more frisbees than deck cards, meaning she has 30-20 = 10 deck cards\n",
      "If she buys 2/5 times more deck cards, she'll have 2/5*10 = 4 more deck cards.\n",
      "The total number of deck cards she'll have is 10+4 = 14\n",
      "Together, Bella will have a total of 14+42+84 = 140 items\n",
      "The answer is 140\n",
      "\n",
      "Question: A group of 4 fruit baskets contains 9 apples, 15 oranges, and 14 bananas in the first three baskets and 2 less of each fruit in the fourth basket. How many fruits are there?\n",
      "Let's think step by step\n",
      "For the first three baskets, the number of apples and oranges in one basket is 9+15=24\n",
      "In total, together with bananas, the number of fruits in one basket is 24+14=38 for the first three baskets.\n",
      "Since there are three baskets each having 38 fruits, there are 3*38=114 fruits in the first three baskets.\n",
      "The number of apples in the fourth basket is 9-2=7\n",
      "There are also 15-2=13 oranges in the fourth basket\n",
      "The combined number of oranges and apples in the fourth basket is 13+7=20\n",
      "The fourth basket also contains 14-2=12 bananas.\n",
      "In total, the fourth basket has 20+12=32 fruits.\n",
      "The four baskets together have 32+114=146 fruits.\n",
      "The answer is 146\n",
      "\n",
      "Question: You can buy 4 apples or 1 watermelon for the same price. You bought 36 fruits evenly split between oranges, apples and watermelons, and the price of 1 orange is $0.50. How much does 1 apple cost if your total bill was $66?\n",
      "Let's think step by step\n",
      "If 36 fruits were evenly split between 3 types of fruits, then I bought 36/3 = 12 units of each fruit\n",
      "If 1 orange costs $0.50 then 12 oranges will cost $0.50 * 12 = $6\n",
      "If my total bill was $66 and I spent $6 on oranges then I spent $66 - $6 = $60 on the other 2 fruit types.\n",
      "Assuming the price of watermelon is W, and knowing that you can buy 4 apples for the same price and that the price of one apple is A, then 1W=4A\n",
      "If we know we bought 12 watermelons and 12 apples for $60, then we know that $60 = 12W + 12A\n",
      "Knowing that 1W=4A, then we can convert the above to $60 = 12(4A) + 12A\n",
      "$60 = 48A + 12A\n",
      "$60 = 60A\n",
      "Then we know the price of one apple (A) is $60/60= $1\n",
      "The answer is 1\n",
      "\n",
      "Question: Susy goes to a large school with 800 students, while Sarah goes to a smaller school with only 300 students.  At the start of the school year, Susy had 100 social media followers.  She gained 40 new followers in the first week of the school year, half that in the second week, and half of that in the third week.  Sarah only had 50 social media followers at the start of the year, but she gained 90 new followers the first week, a third of that in the second week, and a third of that in the third week.  After three weeks, how many social media followers did the girl with the most total followers have?\n",
      "Let's think step by step\n",
      "After one week, Susy has 100+40 = 140 followers.\n",
      "In the second week, Susy gains 40/2 = 20 new followers.\n",
      "In the third week, Susy gains 20/2 = 10 new followers.\n",
      "In total, Susy finishes the three weeks with 140+20+10 = 170 total followers.\n",
      "After one week, Sarah has 50+90 = 140 followers.\n",
      "After the second week, Sarah gains 90/3 = 30 followers.\n",
      "After the third week, Sarah gains 30/3 = 10 followers.\n",
      "So, Sarah finishes the three weeks with 140+30+10 = 180 total followers.\n",
      "Thus, Sarah is the girl with the most total followers with a total of 180.\n",
      "The answer is 180\n",
      "\n",
      "Question: Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He rearranged five of these boxes into packages of six highlighters each and sold them for $3 per package. He sold the rest of the highlighters separately at the rate of three pens for $2. How much profit did he make in total, in dollars?\n",
      "Let's think step by step\n",
      "Sam bought 12 boxes x $10 = $120 worth of highlighters.\n",
      "He bought 12 * 30 = 360 highlighters in total.\n",
      "Sam then took 5 boxes × 6 highlighters/box = 30 highlighters.\n",
      "He sold these boxes for 5 * $3 = $15\n",
      "After selling these 5 boxes there were 360 - 30 = 330 highlighters remaining.\n",
      "These form 330 / 3 = 110 groups of three pens.\n",
      "He sold each of these groups for $2 each, so made 110 * 2 = $220 from them.\n",
      "In total, then, he earned $220 + $15 = $235.\n",
      "Since his original cost was $120, he earned $235 - $120 = $115 in profit.\n",
      "The answer is 115\n",
      "\n",
      "Question: In a certain school, 2/3 of the male students like to play basketball, but only 1/5 of the female students like to play basketball. What percent of the population of the school do not like to play basketball if the ratio of the male to female students is 3:2 and there are 1000 students?\n",
      "Let's think step by step\n",
      "The students are divided into 3 + 2 = 5 parts where 3 parts are for males and 2 parts are for females.\n",
      "Each part represents 1000/5 = 200 students.\n",
      "So, there are 3 x 200 = 600 males.\n",
      "And there are 2 x 200 = 400 females.\n",
      "Hence, 600 x 2/3 = 400 males play basketball.\n",
      "And 400 x 1/5 = 80 females play basketball.\n",
      "A total of 400 + 80 = 480 students play basketball.\n",
      "Therefore, 1000 - 480 = 520 do not like to play basketball.\n",
      "The percentage of the school that do not like to play basketball is 520/1000 * 100 = 52\n",
      "The answer is 52\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(prompt_complex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1204557b-25f8-458c-a95b-3e2db046595c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_answer(pred_str, ans_str):\n",
    "    pattern = '\\d*\\.?\\d+'\n",
    "    pred = re.findall(pattern, pred_str)\n",
    "    if(len(pred) >= 1):\n",
    "        # print(pred_str)\n",
    "        pred = pred[-1]\n",
    "        gold = re.findall(pattern, ans_str)\n",
    "        # print(ans_str)\n",
    "        gold = gold[-1]\n",
    "        return pred == gold\n",
    "    else: return False\n",
    "\n",
    "def parse_pred_ans(filename):\n",
    "    with open(filename) as fd: lines = fd.readlines()\n",
    "    am, a = None, None\n",
    "    num_q, acc = 0, 0\n",
    "    current_mode = 'none'\n",
    "    questions = []\n",
    "    ans_pred = []\n",
    "    ans_gold = []\n",
    "    for l in lines:\n",
    "        if(l.startswith('Q: ')):\n",
    "            if(am is not None and a is not None):\n",
    "                questions.append(q)\n",
    "                ans_pred.append(am)\n",
    "                ans_gold.append(a)\n",
    "                if(test_answer(am, a)):\n",
    "                    acc += 1\n",
    "            current_mode = 'q'\n",
    "            q = l\n",
    "            num_q += 1\n",
    "        elif(l.startswith('A_model:')):\n",
    "            current_mode = 'am'\n",
    "            am = l\n",
    "        elif(l.startswith('A:')):\n",
    "            current_mode = 'a'\n",
    "            a = l\n",
    "        else:\n",
    "            if(current_mode == 'q'): q += l\n",
    "            elif(current_mode == 'am'): am += l\n",
    "            elif(current_mode == 'a'): a += l\n",
    "            else:\n",
    "                raise ValueError(current_mode)\n",
    "                \n",
    "    questions.append(q)\n",
    "    ans_pred.append(am)\n",
    "    ans_gold.append(a)\n",
    "    if(test_answer(am, a)):\n",
    "        acc += 1\n",
    "    print('num_q %d correct %d ratio %.4f' % (num_q, acc, float(acc / num_q)))\n",
    "    return questions, ans_pred, ans_gold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "44e7f8ce-f9e7-4c04-96fa-066c023ba4cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_q = prompt_complex + '\\nQuestion: ' + gsm8k_test[1]['question'] + '\\n'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "54e82439-47e4-4e72-ad44-e479c9db3b1b",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?\n",
      "Let's think step by step\n",
      "Angelo and Melanie think they should dedicate 3 hours to each of the 2 chapters, 3 hours x 2 chapters = 6 hours total.\n",
      "For the worksheets they plan to dedicate 1.5 hours for each worksheet, 1.5 hours x 4 worksheets = 6 hours total.\n",
      "Angelo and Melanie need to start with planning 12 hours to study, at 4 hours a day, 12 / 4 = 3 days.\n",
      "However, they need to include time for breaks and lunch. Every hour they want to include a 10-minute break, so 12 total hours x 10 minutes = 120 extra minutes for breaks.\n",
      "They also want to include 3 10-minute snack breaks, 3 x 10 minutes = 30 minutes.\n",
      "And they want to include 30 minutes for lunch each day, so 120 minutes for breaks + 30 minutes for snack breaks + 30 minutes for lunch = 180 minutes, or 180 / 60 minutes per hour = 3 extra hours.\n",
      "So Angelo and Melanie want to plan 12 hours to study + 3 hours of breaks = 15 hours total.\n",
      "They want to study no more than 4 hours each day, 15 hours / 4 hours each day = 3.75\n",
      "They will need to plan to study 4 days to allow for all the time they need.\n",
      "The answer is 4\n",
      "\n",
      "Question: Mark's basketball team scores 25 2 pointers, 8 3 pointers and 10 free throws.  Their opponents score double the 2 pointers but half the 3 pointers and free throws.  What's the total number of points scored by both teams added together?\n",
      "Let's think step by step\n",
      "Mark's team scores 25 2 pointers, meaning they scored 25*2= 50 points in 2 pointers.\n",
      "His team also scores 6 3 pointers, meaning they scored 8*3= 24 points in 3 pointers\n",
      "They scored 10 free throws, and free throws count as one point so they scored 10*1=10 points in free throws.\n",
      "All together his team scored 50+24+10= 84 points\n",
      "Mark's opponents scored double his team's number of 2 pointers, meaning they scored 50*2=100 points in 2 pointers.\n",
      "His opponents scored half his team's number of 3 pointers, meaning they scored 24/2= 12 points in 3 pointers.\n",
      "They also scored half Mark's team's points in free throws, meaning they scored 10/2=5 points in free throws.\n",
      "All together Mark's opponents scored 100+12+5=117 points\n",
      "The total score for the game is both team's scores added together, so it is 84+117=201 points\n",
      "The answer is 201\n",
      "\n",
      "Question: Bella has two times as many marbles as frisbees. She also has 20 more frisbees than deck cards. If she buys 2/5 times more of each item, what would be the total number of the items she will have if she currently has 60 marbles?\n",
      "Let's think step by step\n",
      "When Bella buys 2/5 times more marbles, she'll have increased the number of marbles by 2/5*60 = 24\n",
      "The total number of marbles she'll have is 60+24 = 84\n",
      "If Bella currently has 60 marbles, and she has two times as many marbles as frisbees, she has 60/2 = 30 frisbees.\n",
      "If Bella buys 2/5 times more frisbees, she'll have 2/5*30 = 12 more frisbees.\n",
      "The total number of frisbees she'll have will increase to 30+12 = 42\n",
      "Bella also has 20 more frisbees than deck cards, meaning she has 30-20 = 10 deck cards\n",
      "If she buys 2/5 times more deck cards, she'll have 2/5*10 = 4 more deck cards.\n",
      "The total number of deck cards she'll have is 10+4 = 14\n",
      "Together, Bella will have a total of 14+42+84 = 140 items\n",
      "The answer is 140\n",
      "\n",
      "Question: A group of 4 fruit baskets contains 9 apples, 15 oranges, and 14 bananas in the first three baskets and 2 less of each fruit in the fourth basket. How many fruits are there?\n",
      "Let's think step by step\n",
      "For the first three baskets, the number of apples and oranges in one basket is 9+15=24\n",
      "In total, together with bananas, the number of fruits in one basket is 24+14=38 for the first three baskets.\n",
      "Since there are three baskets each having 38 fruits, there are 3*38=114 fruits in the first three baskets.\n",
      "The number of apples in the fourth basket is 9-2=7\n",
      "There are also 15-2=13 oranges in the fourth basket\n",
      "The combined number of oranges and apples in the fourth basket is 13+7=20\n",
      "The fourth basket also contains 14-2=12 bananas.\n",
      "In total, the fourth basket has 20+12=32 fruits.\n",
      "The four baskets together have 32+114=146 fruits.\n",
      "The answer is 146\n",
      "\n",
      "Question: You can buy 4 apples or 1 watermelon for the same price. You bought 36 fruits evenly split between oranges, apples and watermelons, and the price of 1 orange is $0.50. How much does 1 apple cost if your total bill was $66?\n",
      "Let's think step by step\n",
      "If 36 fruits were evenly split between 3 types of fruits, then I bought 36/3 = 12 units of each fruit\n",
      "If 1 orange costs $0.50 then 12 oranges will cost $0.50 * 12 = $6\n",
      "If my total bill was $66 and I spent $6 on oranges then I spent $66 - $6 = $60 on the other 2 fruit types.\n",
      "Assuming the price of watermelon is W, and knowing that you can buy 4 apples for the same price and that the price of one apple is A, then 1W=4A\n",
      "If we know we bought 12 watermelons and 12 apples for $60, then we know that $60 = 12W + 12A\n",
      "Knowing that 1W=4A, then we can convert the above to $60 = 12(4A) + 12A\n",
      "$60 = 48A + 12A\n",
      "$60 = 60A\n",
      "Then we know the price of one apple (A) is $60/60= $1\n",
      "The answer is 1\n",
      "\n",
      "Question: Susy goes to a large school with 800 students, while Sarah goes to a smaller school with only 300 students.  At the start of the school year, Susy had 100 social media followers.  She gained 40 new followers in the first week of the school year, half that in the second week, and half of that in the third week.  Sarah only had 50 social media followers at the start of the year, but she gained 90 new followers the first week, a third of that in the second week, and a third of that in the third week.  After three weeks, how many social media followers did the girl with the most total followers have?\n",
      "Let's think step by step\n",
      "After one week, Susy has 100+40 = 140 followers.\n",
      "In the second week, Susy gains 40/2 = 20 new followers.\n",
      "In the third week, Susy gains 20/2 = 10 new followers.\n",
      "In total, Susy finishes the three weeks with 140+20+10 = 170 total followers.\n",
      "After one week, Sarah has 50+90 = 140 followers.\n",
      "After the second week, Sarah gains 90/3 = 30 followers.\n",
      "After the third week, Sarah gains 30/3 = 10 followers.\n",
      "So, Sarah finishes the three weeks with 140+30+10 = 180 total followers.\n",
      "Thus, Sarah is the girl with the most total followers with a total of 180.\n",
      "The answer is 180\n",
      "\n",
      "Question: Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He rearranged five of these boxes into packages of six highlighters each and sold them for $3 per package. He sold the rest of the highlighters separately at the rate of three pens for $2. How much profit did he make in total, in dollars?\n",
      "Let's think step by step\n",
      "Sam bought 12 boxes x $10 = $120 worth of highlighters.\n",
      "He bought 12 * 30 = 360 highlighters in total.\n",
      "Sam then took 5 boxes × 6 highlighters/box = 30 highlighters.\n",
      "He sold these boxes for 5 * $3 = $15\n",
      "After selling these 5 boxes there were 360 - 30 = 330 highlighters remaining.\n",
      "These form 330 / 3 = 110 groups of three pens.\n",
      "He sold each of these groups for $2 each, so made 110 * 2 = $220 from them.\n",
      "In total, then, he earned $220 + $15 = $235.\n",
      "Since his original cost was $120, he earned $235 - $120 = $115 in profit.\n",
      "The answer is 115\n",
      "\n",
      "Question: In a certain school, 2/3 of the male students like to play basketball, but only 1/5 of the female students like to play basketball. What percent of the population of the school do not like to play basketball if the ratio of the male to female students is 3:2 and there are 1000 students?\n",
      "Let's think step by step\n",
      "The students are divided into 3 + 2 = 5 parts where 3 parts are for males and 2 parts are for females.\n",
      "Each part represents 1000/5 = 200 students.\n",
      "So, there are 3 x 200 = 600 males.\n",
      "And there are 2 x 200 = 400 females.\n",
      "Hence, 600 x 2/3 = 400 males play basketball.\n",
      "And 400 x 1/5 = 80 females play basketball.\n",
      "A total of 400 + 80 = 480 students play basketball.\n",
      "Therefore, 1000 - 480 = 520 do not like to play basketball.\n",
      "The percentage of the school that do not like to play basketball is 520/1000 * 100 = 52\n",
      "The answer is 52\n",
      "\n",
      "Question: A robe takes 2 bolts of blue fiber and half that much white fiber.  How many bolts in total does it take?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(prompt_q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "155f530f-ce5b-41c2-b105-74b2543ff8dc",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "ename": "InvalidRequestError",
     "evalue": "This model's maximum context length is 2049 tokens, however you requested 2363 tokens (2235 in your prompt; 128 for the completion). Please reduce your prompt; or completion length.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mInvalidRequestError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/zs/5g2jr6ts4yv4d37n62bpwt0c0000gn/T/ipykernel_81667/2230156833.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m                                     \u001b[0mmax_tokens\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m128\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m                                     \u001b[0mn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m                                     \u001b[0mlogprobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m                                    )\n",
      "\u001b[0;32m~/miniconda3/envs/dl_py3.7/lib/python3.7/site-packages/openai/api_resources/completion.py\u001b[0m in \u001b[0;36mcreate\u001b[0;34m(cls, *args, **kwargs)\u001b[0m\n\u001b[1;32m     29\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 31\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     32\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mTryAgain\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     33\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0mstart\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/dl_py3.7/lib/python3.7/site-packages/openai/api_resources/abstract/engine_api_resource.py\u001b[0m in \u001b[0;36mcreate\u001b[0;34m(cls, api_key, api_base, api_type, request_id, api_version, organization, **params)\u001b[0m\n\u001b[1;32m    104\u001b[0m             \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m             \u001b[0mstream\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstream\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 106\u001b[0;31m             \u001b[0mrequest_id\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequest_id\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    107\u001b[0m         )\n\u001b[1;32m    108\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/dl_py3.7/lib/python3.7/site-packages/openai/api_requestor.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, params, headers, files, stream, request_id)\u001b[0m\n\u001b[1;32m    120\u001b[0m             \u001b[0mrequest_id\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequest_id\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    121\u001b[0m         )\n\u001b[0;32m--> 122\u001b[0;31m         \u001b[0mresp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgot_stream\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_interpret_response\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstream\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    123\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgot_stream\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapi_key\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    124\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/dl_py3.7/lib/python3.7/site-packages/openai/api_requestor.py\u001b[0m in \u001b[0;36m_interpret_response\u001b[0;34m(self, result, stream)\u001b[0m\n\u001b[1;32m    328\u001b[0m             return (\n\u001b[1;32m    329\u001b[0m                 self._interpret_response_line(\n\u001b[0;32m--> 330\u001b[0;31m                     \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstatus_code\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstream\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    331\u001b[0m                 ),\n\u001b[1;32m    332\u001b[0m                 \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/dl_py3.7/lib/python3.7/site-packages/openai/api_requestor.py\u001b[0m in \u001b[0;36m_interpret_response_line\u001b[0;34m(self, rbody, rcode, rheaders, stream)\u001b[0m\n\u001b[1;32m    361\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mstream_error\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;36m200\u001b[0m \u001b[0;34m<=\u001b[0m \u001b[0mrcode\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m300\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    362\u001b[0m             raise self.handle_error_response(\n\u001b[0;32m--> 363\u001b[0;31m                 \u001b[0mrbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrcode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstream_error\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstream_error\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    364\u001b[0m             )\n\u001b[1;32m    365\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mInvalidRequestError\u001b[0m: This model's maximum context length is 2049 tokens, however you requested 2363 tokens (2235 in your prompt; 128 for the completion). Please reduce your prompt; or completion length."
     ]
    }
   ],
   "source": [
    "response = openai.Completion.create(model=\"code-cushman-001\", \n",
    "                                    prompt=prompt_q, \n",
    "                                    temperature=0.5, \n",
    "                                    max_tokens=128,\n",
    "                                    n=2,\n",
    "                                    logprobs=5\n",
    "                                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "14ca497f-efaa-4410-afaf-4f70e1e37605",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Let's think step by step\n",
      "The question states that a robe takes 2 bolts of blue fiber and half that much white fiber.\n",
      "This means that the robe takes 2 bolts of blue fiber and 1 bolt of white fiber.\n",
      "So, the total number of bolts is 2 + 1 = 3 bolts.\n",
      "The answer is 3\n",
      "\n",
      "Question: A company offers a special deal where you can buy any combination of 3 items for $4.  If you can buy 1 of each item for $0.50, how much does it cost to buy 5 of each item?\n",
      "Let's think step by step\n",
      "The question states\n"
     ]
    }
   ],
   "source": [
    "print(response['choices'][0]['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "bf90ffe1-8412-4332-a5d8-fe9dc65d37aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response['choices'][0]['index']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "e7e1a942-5c56-4fd6-9bc5-643f021baaa5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "128"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(response['choices'][0]['logprobs']['top_logprobs'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "783e60fa-51ba-453d-84bd-dcd6cd86e7b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Let's think step by step\n",
      "A robe takes 2 bolts of blue fiber and half that much white fiber.\n",
      "So, a robe takes 2 bolts of blue fiber and 1 bolt of white fiber.\n",
      "So, a robe takes 2 + 1 = 3 bolts in total.\n",
      "The answer is 3\n",
      "\n",
      "Question: A recipe for a cake says that it takes 3 eggs and 4 cups of flour to make a single cake. If you want to make 3 cakes, how many eggs will you need?\n",
      "Let's think step by step\n",
      "A recipe for a cake says that it takes 3 eggs and 4 cups of flour to make\n"
     ]
    }
   ],
   "source": [
    "print(response['choices'][1]['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "12b14767-8dfc-4423-84e7-5a1cffcf6507",
   "metadata": {},
   "outputs": [],
   "source": [
    "ans_model = response['choices'][0]['text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7a921239-a344-4a58-8d25-0460c8a9f3f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_finished(ans_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "67d31467-e02d-4778-8f89-bab6c6a78986",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"Let's think step by step\\nA robe takes 2 bolts of blue fiber and half that much white fiber.\\nHalf of 2 bolts is 1 bolt.\\nSo, a robe takes 2 bolts of blue fiber and 1 bolt of white fiber.\\nThe total number of bolts is 2 + 1 = 3.\\nThe answer is 3\",\n",
       " \"\\nQuestion: A certain number of students are divided into groups of 3.  If there are 2 students left over, how many students are there?\\nLet's think step by step\\nIf there are 2 students left over, then the number of students must be a multiple of 3 plus 2\")"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extract_ans(ans_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "91aa55a3-0f87-4cc7-aa2b-e2168d31819a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_finished(ans_model):\n",
    "    if('answer is' in ans_model): return True\n",
    "    else: return False\n",
    "\n",
    "def extract_ans(ans_model):\n",
    "    ans_model = ans_model.split('\\n')\n",
    "    ans = []\n",
    "    residual = []\n",
    "    for li, al in enumerate(ans_model):\n",
    "        ans.append(al)\n",
    "        if('answer is' in al):\n",
    "            break\n",
    "    residual = list(ans_model[li + 1:])\n",
    "    ans = '\\n'.join(ans)\n",
    "    residual = '\\n'.join(residual)\n",
    "    return ans, residual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "329eaf72-c549-453d-9a4c-7e1ab32400c2",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 15%|██████████████████▍                                                                                                            | 192/1319 [44:46<4:22:50, 13.99s/it]\n"
     ]
    },
    {
     "ename": "RateLimitError",
     "evalue": "Rate limit reached for default-code-davinci-002 in organization org-aXlAUNkHGPgpMHcfe4wtuC5R on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://beta.openai.com/account/billing to add a payment method.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRateLimitError\u001b[0m                            Traceback (most recent call last)",
      "Input \u001b[0;32mIn [37]\u001b[0m, in \u001b[0;36m<cell line: 3>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m prompt_q \u001b[38;5;241m=\u001b[39m prompt_complex \u001b[38;5;241m+\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124mQuestion: \u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;241m+\u001b[39m q \u001b[38;5;241m+\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m      8\u001b[0m call_time_start \u001b[38;5;241m=\u001b[39m time\u001b[38;5;241m.\u001b[39mtime()\n\u001b[0;32m---> 10\u001b[0m response \u001b[38;5;241m=\u001b[39m \u001b[43mopenai\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mCompletion\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcreate\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmodel\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mcode-davinci-002\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\n\u001b[1;32m     11\u001b[0m \u001b[43m                                    \u001b[49m\u001b[43mprompt\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mprompt_q\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\n\u001b[1;32m     12\u001b[0m \u001b[43m                                    \u001b[49m\u001b[43mtemperature\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\n\u001b[1;32m     13\u001b[0m \u001b[43m                                    \u001b[49m\u001b[43mmax_tokens\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m256\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m     14\u001b[0m ans_model \u001b[38;5;241m=\u001b[39m response[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mchoices\u001b[39m\u001b[38;5;124m'\u001b[39m][\u001b[38;5;241m0\u001b[39m][\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtext\u001b[39m\u001b[38;5;124m'\u001b[39m]\n\u001b[1;32m     15\u001b[0m \u001b[38;5;66;03m# if(test_finished(ans_model) == False): \u001b[39;00m\n\u001b[1;32m     16\u001b[0m \u001b[38;5;66;03m#     response = openai.Completion.create(model=\"code-davinci-002\", \u001b[39;00m\n\u001b[1;32m     17\u001b[0m \u001b[38;5;66;03m#                                     prompt=prompt_q, \u001b[39;00m\n\u001b[1;32m     18\u001b[0m \u001b[38;5;66;03m#                                     temperature=0, \u001b[39;00m\n\u001b[1;32m     19\u001b[0m \u001b[38;5;66;03m#                                     max_tokens=256)\u001b[39;00m\n\u001b[1;32m     20\u001b[0m \u001b[38;5;66;03m#     ans_model = response['choices'][0]['text']\u001b[39;00m\n",
      "File \u001b[0;32m~/miniconda3/envs/dl_py3.8/lib/python3.8/site-packages/openai/api_resources/completion.py:25\u001b[0m, in \u001b[0;36mCompletion.create\u001b[0;34m(cls, *args, **kwargs)\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m:\n\u001b[1;32m     24\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 25\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcreate\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     26\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m TryAgain \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m     27\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m timeout \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m time\u001b[38;5;241m.\u001b[39mtime() \u001b[38;5;241m>\u001b[39m start \u001b[38;5;241m+\u001b[39m timeout:\n",
      "File \u001b[0;32m~/miniconda3/envs/dl_py3.8/lib/python3.8/site-packages/openai/api_resources/abstract/engine_api_resource.py:115\u001b[0m, in \u001b[0;36mEngineAPIResource.create\u001b[0;34m(cls, api_key, api_base, api_type, request_id, api_version, organization, **params)\u001b[0m\n\u001b[1;32m    107\u001b[0m requestor \u001b[38;5;241m=\u001b[39m api_requestor\u001b[38;5;241m.\u001b[39mAPIRequestor(\n\u001b[1;32m    108\u001b[0m     api_key,\n\u001b[1;32m    109\u001b[0m     api_base\u001b[38;5;241m=\u001b[39mapi_base,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    112\u001b[0m     organization\u001b[38;5;241m=\u001b[39morganization,\n\u001b[1;32m    113\u001b[0m )\n\u001b[1;32m    114\u001b[0m url \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mcls\u001b[39m\u001b[38;5;241m.\u001b[39mclass_url(engine, api_type, api_version)\n\u001b[0;32m--> 115\u001b[0m response, _, api_key \u001b[38;5;241m=\u001b[39m \u001b[43mrequestor\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrequest\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    116\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mpost\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m    117\u001b[0m \u001b[43m    \u001b[49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    118\u001b[0m \u001b[43m    \u001b[49m\u001b[43mparams\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mparams\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    119\u001b[0m \u001b[43m    \u001b[49m\u001b[43mheaders\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mheaders\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    120\u001b[0m \u001b[43m    \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    121\u001b[0m \u001b[43m    \u001b[49m\u001b[43mrequest_id\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrequest_id\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    122\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    124\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m stream:\n\u001b[1;32m    125\u001b[0m     \u001b[38;5;66;03m# must be an iterator\u001b[39;00m\n\u001b[1;32m    126\u001b[0m     \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(response, OpenAIResponse)\n",
      "File \u001b[0;32m~/miniconda3/envs/dl_py3.8/lib/python3.8/site-packages/openai/api_requestor.py:121\u001b[0m, in \u001b[0;36mAPIRequestor.request\u001b[0;34m(self, method, url, params, headers, files, stream, request_id)\u001b[0m\n\u001b[1;32m    102\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrequest\u001b[39m(\n\u001b[1;32m    103\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m    104\u001b[0m     method,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    110\u001b[0m     request_id: Optional[\u001b[38;5;28mstr\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[1;32m    111\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Tuple[Union[OpenAIResponse, Iterator[OpenAIResponse]], \u001b[38;5;28mbool\u001b[39m, \u001b[38;5;28mstr\u001b[39m]:\n\u001b[1;32m    112\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mrequest_raw(\n\u001b[1;32m    113\u001b[0m         method\u001b[38;5;241m.\u001b[39mlower(),\n\u001b[1;32m    114\u001b[0m         url,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    119\u001b[0m         request_id\u001b[38;5;241m=\u001b[39mrequest_id,\n\u001b[1;32m    120\u001b[0m     )\n\u001b[0;32m--> 121\u001b[0m     resp, got_stream \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_interpret_response\u001b[49m\u001b[43m(\u001b[49m\u001b[43mresult\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    122\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m resp, got_stream, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mapi_key\n",
      "File \u001b[0;32m~/miniconda3/envs/dl_py3.8/lib/python3.8/site-packages/openai/api_requestor.py:328\u001b[0m, in \u001b[0;36mAPIRequestor._interpret_response\u001b[0;34m(self, result, stream)\u001b[0m\n\u001b[1;32m    320\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m (\n\u001b[1;32m    321\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_interpret_response_line(\n\u001b[1;32m    322\u001b[0m             line, result\u001b[38;5;241m.\u001b[39mstatus_code, result\u001b[38;5;241m.\u001b[39mheaders, stream\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[1;32m    323\u001b[0m         )\n\u001b[1;32m    324\u001b[0m         \u001b[38;5;28;01mfor\u001b[39;00m line \u001b[38;5;129;01min\u001b[39;00m parse_stream(result\u001b[38;5;241m.\u001b[39miter_lines())\n\u001b[1;32m    325\u001b[0m     ), \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[1;32m    326\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    327\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m (\n\u001b[0;32m--> 328\u001b[0m         \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_interpret_response_line\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    329\u001b[0m \u001b[43m            \u001b[49m\u001b[43mresult\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcontent\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mresult\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstatus_code\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mresult\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mheaders\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\n\u001b[1;32m    330\u001b[0m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m,\n\u001b[1;32m    331\u001b[0m         \u001b[38;5;28;01mFalse\u001b[39;00m,\n\u001b[1;32m    332\u001b[0m     )\n",
      "File \u001b[0;32m~/miniconda3/envs/dl_py3.8/lib/python3.8/site-packages/openai/api_requestor.py:361\u001b[0m, in \u001b[0;36mAPIRequestor._interpret_response_line\u001b[0;34m(self, rbody, rcode, rheaders, stream)\u001b[0m\n\u001b[1;32m    359\u001b[0m stream_error \u001b[38;5;241m=\u001b[39m stream \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124merror\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m resp\u001b[38;5;241m.\u001b[39mdata\n\u001b[1;32m    360\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m stream_error \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;241m200\u001b[39m \u001b[38;5;241m<\u001b[39m\u001b[38;5;241m=\u001b[39m rcode \u001b[38;5;241m<\u001b[39m \u001b[38;5;241m300\u001b[39m:\n\u001b[0;32m--> 361\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mhandle_error_response(\n\u001b[1;32m    362\u001b[0m         rbody, rcode, resp\u001b[38;5;241m.\u001b[39mdata, rheaders, stream_error\u001b[38;5;241m=\u001b[39mstream_error\n\u001b[1;32m    363\u001b[0m     )\n\u001b[1;32m    364\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m resp\n",
      "\u001b[0;31mRateLimitError\u001b[0m: Rate limit reached for default-code-davinci-002 in organization org-aXlAUNkHGPgpMHcfe4wtuC5R on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://beta.openai.com/account/billing to add a payment method."
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "per_call_sec = []\n",
    "with open('outputs/test_codex_complex.txt', 'w') as fd:\n",
    "    for q, a in tqdm(zip(gsm8k_test['question'], gsm8k_test['answer']), \n",
    "                               total=len(gsm8k_test['question'])):\n",
    "        prompt_q = prompt_complex + '\\nQuestion: ' + q + '\\n'\n",
    "        \n",
    "        call_time_start = time.time()\n",
    "        \n",
    "        response = openai.Completion.create(model=\"code-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        # if(test_finished(ans_model) == False): \n",
    "        #     response = openai.Completion.create(model=\"code-davinci-002\", \n",
    "        #                                     prompt=prompt_q, \n",
    "        #                                     temperature=0, \n",
    "        #                                     max_tokens=256)\n",
    "        #     ans_model = response['choices'][0]['text']\n",
    "        call_time_end = time.time()\n",
    "        per_call_sec.append(call_time_end - call_time_start)\n",
    "        \n",
    "        ans_, residual = extract_ans(ans_model)\n",
    "            \n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_, a))\n",
    "        i += 1\n",
    "        sleep(3)\n",
    "        # if(i == 2): break\n",
    "        # if(i % 10 == 0): \n",
    "        # print('per call sec %.2f' % np.average(per_call_sec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "efef9d01-229f-4b68-8ff7-bb1d9fca80ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 192 correct 123 ratio 0.6406\n"
     ]
    }
   ],
   "source": [
    "_, _, _ = parse_pred_ans('outputs/test_codex_complex.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "a7bb8ed3-5f6d-43b6-b571-0f6092033082",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tenacity import (\n",
    "    retry,\n",
    "    stop_after_attempt,\n",
    "    wait_chain,\n",
    "    wait_fixed\n",
    ") \n",
    "\n",
    "@retry(wait=wait_chain(*[wait_fixed(3) for i in range(3)] +\n",
    "                       [wait_fixed(5) for i in range(2)] +\n",
    "                       [wait_fixed(10)]))\n",
    "def completion_with_backoff(**kwargs):\n",
    "    return openai.Completion.create(**kwargs)\n",
    "\n",
    "def test_from_k(k=0):\n",
    "    i = 0\n",
    "    per_call_sec = []\n",
    "    with open('outputs/test_codex_complex_from_%d.txt' % k, 'w') as fd:\n",
    "        for q, a in tqdm(zip(gsm8k_test['question'][k:], gsm8k_test['answer'][k:]), \n",
    "                                   total=len(gsm8k_test['question'][k:])):\n",
    "            prompt_q = prompt_complex + '\\nQuestion: ' + q + '\\n'\n",
    "\n",
    "            call_time_start = time.time()\n",
    "\n",
    "            response = completion_with_backoff(model=\"code-davinci-002\", \n",
    "                                                prompt=prompt_q, \n",
    "                                                temperature=0, \n",
    "                                                max_tokens=256)\n",
    "            # response = openai.Completion.create(model=\"code-davinci-002\", \n",
    "            #                                     prompt=prompt_q, \n",
    "            #                                     temperature=0, \n",
    "            #                                     max_tokens=256)\n",
    "            ans_model = response['choices'][0]['text']\n",
    "\n",
    "            call_time_end = time.time()\n",
    "            per_call_sec.append(call_time_end - call_time_start)\n",
    "\n",
    "            ans_, residual = extract_ans(ans_model)\n",
    "\n",
    "            fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_, a))\n",
    "            i += 1\n",
    "            sleep(3)\n",
    "            # if(i == 2): break\n",
    "            # if(i % 10 == 0): \n",
    "            # print('per call sec %.2f' % np.average(per_call_sec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "eddda903-3646-41ef-80d3-955d40c83302",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1127/1127 [4:16:51<00:00, 13.67s/it]\n"
     ]
    }
   ],
   "source": [
    "test_from_k(k=192)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "f4adb86e-fc2a-4653-8aa7-7ec54f356396",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 1127 correct 755 ratio 0.6699\n"
     ]
    }
   ],
   "source": [
    "_, _, _ = parse_pred_ans('outputs/test_codex_complex_from_192.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "0e4caea4-23b9-48ca-9dda-30d9ccc9be06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.66565579984837"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(755 + 123) / (1127 + 192)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86ad0c9d-c6e0-4ceb-aaa1-00ac6c97ab71",
   "metadata": {},
   "source": [
    "# Random Prompt, Acc 60.35"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "85b2816e-dd9a-45f2-bc7e-887df14ae0fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_random = open('../gsm8k/lib_prompt/prompt_random.txt').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "52e57bfe-d048-4a0c-9f9e-6e6f256e0485",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [4:02:29<00:00, 11.03s/it]\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "with open('outputs/test_codex_random.txt', 'w') as fd:\n",
    "    for q, a in tqdm(zip(gsm8k_test['question'], gsm8k_test['answer']), \n",
    "                               total=len(gsm8k_test['question'])):\n",
    "        prompt_q = prompt_random + '\\nQuestion: ' + q + '\\n'\n",
    "\n",
    "        response = completion_with_backoff(model=\"code-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        ans_, residual = extract_ans(ans_model)\n",
    "\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_, a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "15cef9fe-acb3-42c5-9f1b-9ec02fc864e2",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Georgie is a varsity player on a football team. He can run 40 yards within 5 seconds. If he can improve his speed by forty percent, how many yards will he be able to run within 10 seconds?\n",
      "Let's think step by step\n",
      "The additional yard he will be able to run to after improving his speed is 40 x 40/100 = 16 yards.\n",
      "So the total yard he will be able to run to after improving his speed is 40 + 16 = 56 yards in 5 seconds.\n",
      "Therefore the total yards he will be able to run to within 10 seconds is 56 x 2 = 112.\n",
      "The answer is 112\n",
      "\n",
      "Question: Hana sold 4/7 of her stamp collection for $28. How much would she have earned from selling the entire collection?\n",
      "Let's think step by step\n",
      "Hana sold 4/7 of her collection for $28, so 1/7 of her collection represents: 28/4 = $7.\n",
      "And as a result, the entire collection represents: 7 * 7 = $49.\n",
      "The answer is 49\n",
      "\n",
      "Question: A shoe company sells $4000 worth of shoes every month. If they want to make $60000 in an entire year how many more dollars per month should they make selling shoes?\n",
      "Let's think step by step\n",
      "The amount of money they currently make in a year is 12 * 4000 = $48000.\n",
      "The additional amount they need to make is $60000 - $48000= $12000 per year.\n",
      "So, they should make $12000 / 12 = $1000 more every month.\n",
      "The answer is 1000\n",
      "\n",
      "Question: Olga has an aquarium with fish in 5 different colors. There are 12 yellow ones, half as many blue ones, and twice as many green ones as yellow ones. How many fish does Olga have in her aquarium?\n",
      "Let's think step by step\n",
      "Olga has half as many blue fish as yellow ones, which means 12 * 0.5 = 6 blue fish.\n",
      "Green fish are twice as many as yellow ones, so in total there are 12 * 2 = 24 green fish.\n",
      "So all the fish sum up to 12 + 6 + 24 = 42.\n",
      "The answer is 42\n",
      "\n",
      "Question: Every time Carl earned $0.50 he would go to the corner market and buy a candy bar.  Carl's neighbor said he would pay him $0.75 every week for taking out his trash.  At the end of four weeks, how many candy bars will Carl be able to buy?\n",
      "Let's think step by step\n",
      "He is paid $0.75 every week for 4 weeks, so Carl made .75 * 4 = $3.00 in 4 weeks\n",
      "Each candy bar cost $0.50 and he made $3.00 so 3.00/.50 = 6 candy bars\n",
      "The answer is 6\n",
      "\n",
      "Question: Oranges have 80 calories and cost $1.20 each. If Timmy has $10 and needs to make sure he gets 400 calories, how much money will he have left after he buys the oranges he needs?\n",
      "Let's think step by step\n",
      "He needs to buy 5 oranges because 400 / 80 = 5\n",
      "These will cost $6 because 5 x 1.2 = 6\n",
      "He will have $4 left because 10 - 6 = 4\n",
      "The answer is 4\n",
      "\n",
      "Question: A school does a yearly tree planting activity. This year, the 4th graders planted a total of 30 trees while 5th graders planted twice as many. The 6th graders planted 30 fewer than thrice the number of trees planted by the 5th graders. How many trees did they plant altogether?\n",
      "Let's think step by step\n",
      "The 5th graders planted 30 x 2 = 60 trees.\n",
      "Thrice the number of trees planted by the 5th graders is 60 x 3 = 180.\n",
      "So, the 6th graders planted 180 - 30 = 150 trees.\n",
      "Therefore, they planted a total of 30 + 60 + 150 = 240 trees.\n",
      "The answer is 240\n",
      "\n",
      "Question: Jonah decided to set up an aquarium. He started with 14 small fish. He added 2 more, but they ate 6 of his original fish before he could remove them and take them back to the store. He exchanged them for 3 new fish. How many fish does Jonah have now?\n",
      "Let's think step by step\n",
      "Jonah started with 14 fish + 2 more = 16 fish.\n",
      "Of the 16 fish – 6 were eaten – 2 were returned to the store = 8 fish.\n",
      "After he returned the fish to the store, he exchanged them for 3 new fish + 8 fish that were left = 11 fish now in his tank.\n",
      "The answer is 11\n",
      "\n",
      "Question: James decides to run 3 sprints 3 times a week.  He runs 60 meters each sprint.  How many total meters does he run a week?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(prompt_q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "c54fcaee-188e-47f9-846d-3407f9cad6d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 1319 correct 796 ratio 0.6035\n"
     ]
    }
   ],
   "source": [
    "_, _, _ = parse_pred_ans('outputs/test_codex_random.txt')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
